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1.0  MODES  DESIGN  REQUIREMENTS 


Central  to  any  validation  of  MODES  are  the  principal 
requirements  under  which  it  was  designed  and  constructed*. 

(1)  MODES  should  support  crisis  action  deployment 
planning . 

i 

<E)  MODES  should  aid  in  the  very  early  part  of  the 
crisis  action  planning  process. 

<3)  MODES  should  provide  useful  information  in  two  to 
four  hours,  f 

) 

(4)  MODES  should  address  questions  concerning  "gross 
transportation  feasibi  1  ity1*' among  alternative  courses  of 
action. 

A  comprehensive  understanding  of  these  design  criteria  is 
essential  in  developing  a  reasonable  validation  process. 


1 . 1  Supporting  Cr i.sis  Action  Planning 

The  operative  terms  here  are  “support"  and  "crisis  action". 
MODES  was  not  designed  to  replace  the  Supported  Commander 
and  his  staff.  It  was  intended  as  a  tool  for  enhancing 
their, capabi 1 i ties.  As  a  result  MODES  should  be  evaluated 
with  planners,  not  against  them. 

MODES  is  not  a  deliberate  planning  tool.  It  was  not 
designed  to  accomodate  the  level  of  detail  which  is  possible 
in  a  deliberate  planning  cycle.  It  was  designed  to  aid  in 
making  rapid  go/nogo  decisions  at  the  first  level  in  a 
hierarchical  planning  system.  At  its  level  of  use,  it 
should  help  establish  attainable  EADs,  LADs,  and  RDDs  as 
well  as  provide  recommendations  on  channel  size,  on  mode, 
and  on  general  routing  for  movement  requirements.  It  should 
also  provides  gross  feasibi ity  assessments  with  regard  to 
closure.  MODES  was  not  designed  to  provide  the  detail 
scheduling  information  currently  generated  in  the  deliberate 
planning  process.  The  output  of  MODES  should  be  viewed  as 
one  of  the  inputs  to  the  detailed  scheduling  process. 
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1.2  Initial  Planning  Capability 

MODES  was  designed  to  provide  initial  planning  capability  to 
the  Supported  Commander  and  his  staff  in  the  very  early 
stages  of  the  crisis  action  planning  process.  As  a  result, 
MODES  must  be  prepared  to  work  with  a  variety  of  levels  and 
quality  of  data  in  providing  information  on  various  courses 
of  action. 


1.3  S§E!S  TyEDaround 

One  of  the  strongest  design  criteria  for  MODES  was  that  it 
must  provide  information  on  reasonable  sized  courses  of 
action  in  two  to  four  hours.  This  criteria  had  the  single 
greatest  effect  on  the  final  design  for  MODES. 


1.4  Gross  Measures  of  Transportation  Feasibility 

MODES  was  designed  to  provide  estimates  of  gross 
transportation  feasibility  to  the  Supported  Commander.  Once 
MODES  provides  gross  feasibility  estimates >  at  least  two 
more  levels  of  analyses  are  required  to  generate  the 
detailed  schedules  and  corresponding  detailed  closure 
estimates  required  by  the  TOAs.  Models  to  support  these 
additional  levels  of  analysis  are  currently  being  actively 
developed  by  the  TOAs. 


2.0  A  STRUCTURE  FOR  MODES  VALIDATION 

With  the  MODES  design  criteria  in  mind»  we  believe  that  a 
reasonable  MODES  validation  process  should  try  to  answer  the 
following  three  questions: 

(1)  Are  the  MODES  recommendations  reasonable? 

<2)  What  MODES  resolution  is  practical  in  reasonable 
response  time? 

(3)  What  is  the  quality  of  the  MODES  recommendations? 

Each  of  these  questions  and  associated  analyses  will  be 
discussed  in  turn. 


2.1  iiity 

Reasonability  analysis  deals  with  the  ability  of  MODES  to 
provide  results  which  describe  reasonable  outputs  for  given 
inputs.  In  a  sense*  this  is  an  intermediate  step  between 
verification  analysis  and  final  validation. 


2.1.1  Extremal  Analysis 


In  testing  a  procedure  for  reasonableness  it  is  generally 
easier  to  test  the  extremes  before  testing  mid-range 
results.  In  an  extremal  analysis  of  MODES ,  a  number  (ten  to 
twenty)  of  scenarios  should  be  described  with  the  parameters 
fixed  so  that  the  only  reasonable  MODES  outputs  are  readily 
predicted  by  analysts.  Examples  include  small  scenarios 
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with  fixed  ports*  fixed  channels*  and/or  limited  assets. 

Each  resulting  MODES  output  should  then  be  evaluated  by 
knowledgeable  planners  for  reasonableness  of  response  to  the 
constraining  input  conditions.  Each  evaluation  would  be 
rated  "pass11  or  "fail".  Failures  would  be  acompanied  by  an 
explanation  of  the  difficulty.  Each  failure  or  difficulty 
would  be  reviewed  by  MODES  designers  for  possible 
explanation  or  subsequent  MODES  modification  and  adjustment. 

Extremal  analyses  test  the  boundaries  of  MODES  responses  to 
assure  valid  results  in  extraordinary  situations. 


2.1.2  Parametric  Analysis 

The  next  step  in  a  reasonability  analysis  is  to  evaluate 
MODES  responses  to  variations  in  parameters.  In  this 
analysis*  more  involved  scenarios  should  be  tested  wherein 
MODES  results  are  not  so  obviously  predictable.  The  output 
should  be  evaluated  by  a  team  of  planners.  The  team  would 
attempt  to  rationalize  the  MODES  output  or  to  point  out 
inadequacies  in  the  logic. 

In  this  analysis  the  MODES  parameters  would  be  varied  over  a 
range  of  values.  The.  objective  is  to  test  sensitivity  of 
parameter  values  to  MODES  output  responses.  (This  also 
serves  as  a  "tuning"  of  the  MODES  system.)  Each  report  (by 
the  test  team)  would  contain  an  analysis  of  the  robustness 
of  the  MODES  model  to  help  define  rational  responses  to 
varying  threats. 


2.1.3  Feasibility  Analysis 

This  analysis  would  help  answer  the  question  of  "what  is 
reasonable  MODES  resolution?" 

This  analysis  is  accomplished  by  first  applying  MODES  to  a 
set  of  test  scenarios  and  then  using  the  MODES  output  as 
input  to  the  detailed  scheduling  process.  The  detailed 
scheduling  could  then  either  be  performed  manually  or  for 
larger  scenarios  with  the  aid  of  TFE  or  the  TOA  scheduling 
models.  The  detailed  schedules  and  corresponding  detailed 
closure  estimates  should  then  be  compared  with  the  output  of 
MODES  to  determine  overall  consistency.  Note  that  this  is 
not  the  same  as  comparing  MODES  generated  results  with 
simulation  generated  results.  Such  a  comparison  would  be 
meaningless  since  the  models  do  not  utilize  the  same  levels 
of  detail  and  the  simulation  models  themselves  have  never 
been  validated.  The  test  described  here  would  test  whether 
given  levels  of  resolution  are  consistent  with  simulation 
models  in  reporting  port/asset  loadings  and  closure 
estimates. 
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K  This  testing  sequence  would  be  repeated  for  varying 

™  resolution  of  the  MODES  model  (ports;  assets;  time).  A 

panel  of  planners  would  evaluate  MODES  output  with  the 
resulting  simulation  results  and  submit  a  report  comparing 
and  contrasting  the  results.  The  idea  here  is  not  to  charge 
MODES  with  the  responsibility  of  tracking  simulation 
results;  but  to  use  simulation  results  as  a  means  for 
^  experienced  planners  to  assess  relative  precision  of 

‘X  results.  This  will  provide  experienced  planners  with  a 

framework  for  making  reasonable  judgements  of  desired 
resolution.  It  will  also  indicate  whether  the  input 
•-?  parameters  for  MODES  need  to  be  factored  to  account  for  the 

loss  of  detail  in  the  data.  For  example;  it  may  be 
desirable  to  have  MODES  work  under  the  assumption  that  less 
than  one  hundred  percent  of  the  capability  of  a  channel  is 
actually  usable. 
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3.0  RESPONSE  TIME  ANALYSIS 

In  this  series  of  analyses  various  sizes  of  problems  are 
input  to  the  MODES  model  for  solution.  By  varying  numbers 
of  ports;  asset  types;  movement  requirements;  and 
aggregation  levels;  and  charting  the  resulting  solution 
times  for  the  MODES  model;  an  understanding  of  the  response 
time  of  the  MODES  model  under  varying  problem  conditions 
will  be  obtained. 


3 . 1  Parameter  Selection 

Not  all  parameters  affect  the  MODES  model  in  the  same 
manner.  Adding  asset  categories  increases  the  LIFTCAP 
problem  only  slightly;  while  leaving  the  size  of  the  MRMATE 
problem  unchanged.  Adding  ports  increases  the  LIFTCAP 
problem  to  a  greater  extent  and  the  MRMATE  problem 
moderately.  Adding  movement  requirements  adds  only  to  the 
MRMATE  problem. 

The  LIFTCAP  problem  generaly  requires  much  less  time  to 
solve  than  the  MRMATE  problem.  Therefore  any  parameter 
change  which  significantly  affect  the  size  of  the  MRMATE 
problem  will  generally  result  in  longer  solution  times. 


3.E  Aggregation  Level 

The  single  most  significant  determinant  of  problem  running 
time  is  aggregation  level.  A  slight  change  in  aggregation 
level  of  almost  any  of  the  variables  can  cause  the  MRMATE 
problem  to  change  considerably  in  size  and  produce 
unacceptable  solution  times. 
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3.3  Predicting  Problem  Size 


It  is  possible  to  predict  problem  size  given  input  numbers 
of  ports)  asset  categories*  time  periods*  and  movement 
requirements.  Georgia  Tech  PDRC  Report  84-09  gave  a  series 
of  formulas  to  predict  LIFTCAP  and  MRMATE  problem  size  given 
input  data.  These  formulas  should  be  helpful  in  judging 
scenario  size  for  testing  responsiveness  of  the  MODES  model. 


4.0  QUALITY  ANALYSIS 

Once  the  limits  of  the  MODES  model  have  been  determined  and 
its  resolution  and  robustness  evaluated*  the  final  test  of 
its  quality  and  usefulness  can  be  made.  This  series  of 
tests  should  evaluate  the  MODES  model  in  planning  situations 
pitted  against  currently  available  planning  tools. 

4 . 1  Control  Grougs 

The  only  reasonable  test  of  MODES  is  in  its  ablitiy  to 
support  crisis  action  planning  in  the  first  two  to  four 
hours.  Therefore*  tests  must  be  developed  to  evaluate  this 
capability.  One  such  method  would  employ  control  groups. 
Teams  of  experienced  planners  would  be  divided  into  two 
groups  and  given  the  same  scenerio  to  evaluate  in  a  two  to 
four  hour  time  limit.  One  group  would  be  permitted  to  use 
MODES  while  the  other  (the  control  group)  would  not.  A 
number  of  scenarios  should  be  evaluated  with  each  group 
having  a  opportunity  to  become  the  control  group  (thus 
minimizing  group  bias).  Also*  if  time  permits  the  groups 
should  be  reconfigured  to  minimize  any  possibility  of 
individual  bias. 

4 . 2  Scenario  Planning 

The  MODES  group  and  the  control  group  should  both  be 
presented  with  identical  scenarios  simulating  a  crisis 
action  planning  situation.  A  time  limit  of  two  to  four 
hours  should  be  given.  Both  groups  may  use  any  technique 
available*  except  that  the  MODES  group  must  also  use  MODES 
and  the  control  group  may  not  use  MODES.  At  the  end  of  the 
alloted  time  both  groups  must  provide  a  recommendation  and 
supporting  analyses. 

This  testing  technique  should  be  repeated  for  several 
scenarios  of  varing  size  and  complexity.  In  some  cases  even 
though  both  groups  must  hand  in  their  recommendations  after 
the  alloted  time*  they  may  continue  planning  for  some  longer 
time  specified  in  the  analysis.  A  second  set  of 
recommendations  would  be  provided  by  each  group  at  the  end 


of  the  new  time  limit.  This  would  give  an  indication  of  the 
value  of  MODES  when  time  is  not  as  critical  and  would 
indicate  the  potential  for  using  MODES  in  the  deliberate 
planning  process. 

4.3  Plan  Evaluation 

For  each  scenario?  boths  sets  of  plans  (MODES  and  control) 
should  be  evaluated  by  a  third  panel  of  experienced 
planners.  The  panel  would  assess  the  relative  quality  of 
the  two  plans.  The  panel  would  not  be  time  constrained  and 
could  use  any  technique  are  analysis  available  to  them  to 
draw  their  conclusions. 

The  panel  would  be  required  to  put  the  two  recomendat ions  on 
a  ordinal  scale  (indicating  which  is  better)  or  on  a 
cardinal  scale  (indicating  how  much  better  one  is  over  the 
other).  An  example  of  a  simple  cardinal  scale  is  "superior? 
better?  equal." 

The  panel  should  also  provide  a  written  assessment  of  the 
two  recommendations  and  supporting  analyses. 

4.4  MODES  Quality  Assessment 

Panel  results  would  be  charted?  and  these  together  with  the 
written  evaluations  provide  a  basis  for  a  validation  of 
MODES.  Some  group  must  evaluate  the  panel  results  and 
results  of  the  earlier  studies  to  write  a  report  on  the 
validation  of  the  MODES  model.  In  the  final  analysis  the 
resulting  judgements  will  be  subjective?  but  this 
subjectivity  will  be  minimized  by  the  intermediate  reports 
and  panel  evaluations. 


